LINEAR REGRESSION MODEL¶
since the targert variable is numeric we use linear regression model
problem statement¶
An insurance company wants to predict the customer life time value on diffrent qualitative and qauntitaive features provided.
They are operating from last few years and maintaining all transactional information data. The given data ‘CustomerData.csv’ is a sample of customer level data extracted and processed for the analysis from various set of transactional files.
OBJECTIVE¶
- client wants to predict the customer life time value building a linear regression model.
steps to calculate customer lifetime value¶
calculate average purchase value
calculate average purchase frequency rate
calculate customer value
calculate average purchase frequency rate
Calculate average customer lifespan
calculate CLTV
how machine learning can help to improve clv¶
A quality management team within an automaker moves with a challenging sphere of tension between customer satisfaction level, regulation and cost control
predictive model can be built to automating customer lifetime value process, hence reducing the unfairness within the clv
loading the required libraries.¶
In [13]:
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
#### Load the required libraries
%matplotlib inline
import os
import inspect
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import SVG
from IPython.display import display
import graphviz
import seaborn
from sklearn.tree import DecisionTreeClassifier
import pandas as pd#(for dataframes)
import numpy as np#(linear algebra)
from sklearn.model_selection import train_test_split#(for spliting the data)
from sklearn.impute import SimpleImputer#(for handling NA's values)
from sklearn.preprocessing import StandardScaler#(it helps in calculation of num data)
#required libraries for visulaization
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline
sns.set_palette("GnBu_d")
sns.set_style('whitegrid')
import missingno as msno
import pandas_profiling as pp
#There is an indication given in the result that there might exist a strong multicollinearity in the data.
#Lets use variance inflation factor (VIF) to understand if there exist a multicollinearity and remove those attributes
from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.preprocessing import LabelEncoder
enc=LabelEncoder()
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier, export_graphviz,DecisionTreeRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import GradientBoostingClassifier
import pandas as pd
import numpy as np
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import seaborn as sns
import math
import statsmodels.api as sm
from statsmodels.formula.api import ols
import pandas as pd
from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules
#mlxtend : Machine learning extensions
import random
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import networkx as nx
## import libraries
import pandas as pd
from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules
#mlxtend : Machine learning extensions
import random
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import networkx as nx
from mlxtend.preprocessing import TransactionEncoder
from mlxtend.frequent_patterns import apriori
## Extract the rules
from mlxtend.frequent_patterns import association_rules
!pip install mlxtend
!pip install networkx
from sklearn.metrics import accuracy_score, recall_score, precision_score,confusion_matrix,mean_absolute_error,mean_squared_error,classification_report
In [14]:
# since pip install graphviz was unable to load this is the extended process to load the library
! pip install graphviz
! pip install mlxtend
In [15]:
! pip install missingno
! pip install pandas_profiling
loading the data¶
1.setting with current working directory with jupyter notebook
In [16]:
# to check which directory my python note book is working
os.getcwd()
os
Out[16]:
In [17]:
# chdir is to change my working directory to my current working directory
os.chdir('C:\\Users\\smithika\\Desktop\\final mith')
In [18]:
# loading the data
df= pd.read_csv("train-1574429526318.csv")
In [19]:
# the detail data which is loaded
df
Out[19]:
Data understanding¶
In [20]:
# data.head gives us the head of data to understand the parameters easy
df.head(5)
Out[20]:
In [21]:
## to check the type of the data
type(df)
Out[21]:
In [22]:
# to check the shape of the data
df.shape
Out[22]:
summary statistics¶
In [23]:
# this is the summary statistic of df
df.describe(include='all')
Out[23]:
In [24]:
df.loc[ (df['number.of.open.complaints']==5.000000),]
In [25]:
df.loc[ (df['number.of.policies']==9.000000),]
In [26]:
#converting all the column names to lower case for more convinience..
df.columns = [x.lower().replace(' ','_') for x in df.columns]
In [27]:
## to check the index of the df
df.index
Out[27]:
In [28]:
# checking wheather names are been converted or not
df.head()
Out[28]:
In [29]:
#writing a function to get the null values and unique values in the dataset
def levels(df):
return (pd.DataFrame({'dtype':df.dtypes,
'levels':df.nunique(),
'levels':[df[x].unique() for x in df.columns],
'null_values':df.isna().sum(),
'unique':df.nunique()}))
levels(df)
Out[29]:
Observations:¶
- we have three diffrent tyes of data i.e object type(which are categorical), Integer64, & float64
- we canot build a model keeping object until is correctly framed with required transformation
- Summary statistics for CustomerID & location.geo is not meaningful hence dropping the columns.
- now we can segregate categorical and numerical attributes separetly.
In [30]:
df.drop("location.geo",axis=1,inplace=True)
df.drop("customerid",axis=1,inplace=True)
In [31]:
df.head(3)
Out[31]:
missing values.¶
In [32]:
## to checking the total missing values
df.isna().sum()
Out[32]:
In [33]:
# dropping the missing values helps us to plot the numeric attributes easily
df.dropna(axis=0, inplace=True)
In [34]:
# checking if we any duplicate values
df2=df.duplicated(keep='first')
In [35]:
df2
Out[35]:
In [36]:
print(df.shape)
In [37]:
## to check the numeric attributes in my df
num_attr = df.select_dtypes(include=['int64', 'float64']).columns
num_attr
Out[37]:
In [38]:
## to check my categorical or object is been identified in df
num_cat= df.select_dtypes(include=['object']).columns
num_cat
Out[38]:
In [39]:
df[num_attr].corr()
Out[39]:
scatter plot matrix required for numeric attributes.¶
In [40]:
pd.plotting.scatter_matrix (df, figsize=(16, 16), diagonal='kde')
plt.show()
In [41]:
plt.figure(figsize=(8,8))
sns.heatmap(df.corr())
plt.show()
exploratory data¶
In [42]:
###Plotting Categorical Data
sns.countplot(x="vehicle.class", data=df)
plt.show()
observation¶
- since we found huge customers who bought the four-door-cars, suv,two-door-car and so on following
- now we can predict the clients who are most bought
converstion of object attributes..& dumification¶
In [43]:
cat=('coverage', 'education', 'employmentstatus', 'gender', 'income',
'location.geo', 'location.code', 'marital.status', 'policy.type',
'policy', 'renew.offer.type', 'sales.channel', 'vehicle.class')
In [44]:
cat
Out[44]:
In [45]:
for col in ['coverage','education','employmentstatus','gender','income',
'location.code',
'marital.status',
'policy.type',
'policy',
'renew.offer.type',
'sales.channel',
'vehicle.class']:
df[col] = df[col].astype('category')
In [46]:
df[col]
Out[46]:
In [47]:
cols=["coverage", "education","employmentstatus","gender","income",
"marital.status",
"policy.type",
"policy",
"renew.offer.type",
"sales.channel",
"vehicle.class"]
df=pd.get_dummies(columns=cols,data=df,drop_first=True)
In [48]:
df.head(3)
Out[48]:
In [49]:
df.isna().sum()
Out[49]:
even after dropping the missing values the location code and monthly premium attribute contained missing values so just droppedit¶
In [50]:
df.dropna(axis=0, inplace=True)
In [51]:
df.isna().sum()
Out[51]:
In [52]:
df.dtypes
Out[52]:
showing the correlation between the target variable with diffrent variables¶
In [53]:
# defining the numerical attributes with value counts
df['vehicle.class'].value_counts()
In [54]:
df['vehicle.size'].value_counts()
Out[54]:
In [68]:
sns.countplot(x="vehicle.size", data=df)
plt.show()
In [69]:
df['renew.offer.type'].value_counts()
In [70]:
df['sales.channel'].value_counts()
In [71]:
df['policy'].value_counts()
In [72]:
df['employmentstatus'].value_counts()
In [73]:
df['policy.type'].value_counts()
In [74]:
sns.boxplot(x="vehicle.class", y="customer.lifetime.value", data=df, palette="PRGn")
plt.show()
In [75]:
sns.boxplot(x="employmentstatus", y="customer.lifetime.value", data=df, palette="PRGn")
plt.show()
In [76]:
# Plotting Categorical Data
sns.countplot(x="customer.lifetime.value", data=df)
plt.show()
In [77]:
# Make default histogram of FrequencyOFPlay
sns.distplot(df["total.claim.amount"] )
# Bins can be changes
plt.show()
In [78]:
#polt histogram of FrequencyOFPlay
sns.distplot(df["customer.lifetime.value"] )
# Bins can be changes
plt.show()
In [79]:
df['customer.lifetime.value'].value_counts()
Out[79]:
In [80]:
# sns plot graph
sns.lmplot(x='total.claim.amount', y='customer.lifetime.value', data=df)
plt.show()
spilting the train and test data¶
In [81]:
#Performing train test split on the data
X, y = df.loc[:,df.columns!='customer.lifetime.value'], df.loc[:,'customer.lifetime.value']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=123)
In [82]:
#To get the distribution in the target in train and test
print(pd.value_counts(y_train))
print(pd.value_counts(y_test))
In [83]:
# loading the train model
train_data = pd.read_csv("train-1574429526318.csv")
train_data.head()
Out[83]:
In [84]:
# understanding the shape of the data
train_data.shape
Out[84]:
In [85]:
## lets us understand the diffrent levels of train data
def understanding_data(data):
return pd.DataFrame({"Data Type":data.dtypes,"No of Levels":data.apply(lambda x: x.nunique(),axis=0), "Levels":data.apply(lambda x: str(x.unique()),axis=0)})
understanding_data(train_data)
Out[85]:
In [86]:
#summary statistics of train data
train_data.describe()
Out[86]:
In [87]:
pp.ProfileReport(train_data)
Out[87]:
In [88]:
temp=train_data.corr()
fig = plt.figure(figsize=(10,8))
sns.heatmap(temp,annot=True)
Out[88]:
In [89]:
# frequency distribution of the data
train_data.hist(bins=50, figsize=(20,20))
plt.show()
In [90]:
# checking for missing data
train_data.isnull().sum()
Out[90]:
In [91]:
#plot for missing data in matrix form
msno.matrix(train_data)
Out[91]:
In [92]:
train_data.dtypes
Out[92]:
In [93]:
#data type conversion
num_cols = ['customer.lifetime.value','monthly.premium.auto','number.of.open.complaints','number.of.policies','total.claim.amount','vehicle.size']
cat_cols = train_data.columns.difference(num_cols)
cat_cols
Out[93]:
In [94]:
#data type conversion
num_cols = ['Customer.Lifetime.Value','Monthly.Premium.Auto','Number.of.Open.Complaints','Number.of.Policies','Total.Claim.Amount','Vehicle.Size']
cat_cols = train_data.columns.difference(num_cols)
cat_cols
Out[94]:
In [98]:
cat_cols
Out[98]:
In [99]:
num_data = train_data.loc[:,num_cols]
cat_data = train_data.loc[:,cat_cols]
In [100]:
train_data[cat_cols] = train_data[cat_cols].apply(lambda x: x.astype('category'))
train_data[num_cols] = train_data[num_cols].apply(lambda x: x.astype('float'))
train_data.dtypes
Out[100]:
In [102]:
# Numeric columns imputation
imp = SimpleImputer(missing_values=np.nan, strategy='median')
num_data = pd.DataFrame(imp.fit_transform(num_data),columns=num_cols)
# Categorical columns imputation
imp = SimpleImputer(missing_values=np.nan, strategy='most_frequent')
cat_data = pd.DataFrame(imp.fit_transform(cat_data),columns=cat_cols)
print(num_data.isnull().sum())
print(cat_data.isnull().sum())
In [103]:
train_data.drop(['CustomerID'], axis=1, inplace=True)
In [104]:
standardizer = StandardScaler()
standardizer.fit(num_data)
num_data = pd.DataFrame(standardizer.transform(num_data),columns=num_cols)
train_data = pd.concat([num_data,cat_data],axis=1)
In [105]:
train_data = pd.get_dummies(train_data,columns=cat_cols,drop_first=True)
In [106]:
train_data.head()
Out[106]:
In [107]:
x = train_data.copy().drop("Customer.Lifetime.Value",axis=1)
y = train_data["Customer.Lifetime.Value"]
In [108]:
x_train, x_validation, y_train, y_validation = train_test_split(x, y, test_size=0.30,random_state=1)
In [109]:
print(train_data.shape)
print(x_train.shape)
print(y_train.shape)
In [ ]:
In [110]:
### Simple Linear Regression using Statsmodels package
##To get all except the target ant target
x.columns
print(x.columns.values[-1])
print(x.columns.values[:-1])
In [111]:
dtr = DecisionTreeRegressor()
dtr.fit(x_train,y_train)
Out[111]:
In [112]:
## Get the predictions on train and test
pred_train = dtr.predict(x_train)
pred_val = dtr.predict(x_validation)
In [113]:
pred_train
Out[113]:
In [114]:
pred_val
Out[114]:
In [115]:
dtr = DecisionTreeRegressor(max_depth=2)
dtr.fit(x_train,y_train)
pred_train = dtr.predict(x_train)
print("Train Error:", mean_absolute_error(y_train,pred_train))
In [117]:
test_df= pd.read_csv("test-1574429501088.csv")
In [118]:
test_df.head(3)
Out[118]:
In [124]:
test_df.describe()
Out[124]:
In [136]:
from sklearn.model_selection import GridSearchCV
def model_building(model, params = None, k = 1) :
if params == None :
model.fit(X_train, y_train)
# return fitted model & train-test predictions
return (model, model.predict(X_train), model.predict(X_test))
else :
model_cv = GridSearchCV(model, param_grid = params, cv = k)
model_cv.fit(X_train, y_train)
model = model_cv.best_estimator_
# return and extra object for all cross validation operations
return (model_cv, model, model.predict(X_train), model.predict(X_test))
### Load the required libraries
%matplotlib inline
import inspect
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import SVG
from IPython.display import display
from sklearn.model_selection import train_test_split
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
num_imputer = SimpleImputer()
num_scaler = StandardScaler()
import seaborn
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.tree import DecisionTreeClassifier, export_graphviz,DecisionTreeRegressor
from sklearn.metrics import accuracy_score, recall_score, precision_score,confusion_matrix,mean_absolute_error,mean_squared_error,classification_report
In [140]:
X_actual_test = test_df
num_imputer.fit(X_actual_test)
num_scaler.fit(X_actual_test)
In [141]:
X_actual_test = num_imputer.transform(X_actual_test)
X_actual_test = num_scaler.transform(X_actual_test)
X_actual_test = pd.DataFrame(X_actual_test)
In [142]:
X_actual_test.isna().sum()
Out[142]:
In [143]:
actual_test_pred = clf.predict(X_actual_test)
In [144]:
actual_test_pred
In [145]:
new_pred = pd.DataFrame(actual_test_pred)
In [147]:
sample= pd.read_csv('sample_submission-1577482703002.csv')
In [151]:
sample.customerLifetime.Value.head()
In [152]:
sample.target=actual_test_pred
In [ ]: